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Abstract 

This  paper  discusses  certain  connections  between  nonlinear  programming  algorithms 
and  the  formulation  of  optimization  problems  for  systems  governed  by  state  constraints. 
The  major  points  of  this  paper  are  the  detailed  calculation  of  the  sensitivities  associated 
with  different  formulations  of  optimization  problems  and  the  identification  of  some  useful  re¬ 
lationships  between  different  formulations.  These  relationships  have  practical  consequences; 
if  one  uses  a  reduced  basis  nonlinear  programming  algorithm,  then  the  implementations  for 
the  different  formulations  need  only  differ  in  a  single  step. 
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1.  Introduction 


This  paper  discusses  certain  useful  relationships  between  nonlinear  programming  algorithms 
and  the  formulation  of  optimization  problems  for  systems  governed  by  state  constraints.  The 
simplest  instance  of  such  a  problem  is 

(1)  minimize  /(x)  = /(a:,u(x)), 

where  /  is  a  real-valued  function  and  u(x)  is  computed  via  the  solution  of  a  system  of 
equations 


h(x,  u(x))  =  0. 


Think  of  X  as  the  design  variables  of  the  problem,  and  of  u  as  the  state  variables  describing 
the  physical  state  of  the  system  to  be  optimized.  The  equations  h(x,  w(x))  =  0  represent 
typically  a  system  of  ODE  or  PDE  that  govern  the  system.  Given  a  value  of  the  design 
variables  x,  we  can  solve  these  equations  for  the  state  u(x).  In  the  computational  setting, 
the  design  variables  x  and  state  variables  u  will  each  lie  in  some  finite-dimensional  vector 
space. 

More  generally,  of  course,  the  problem  could  have  additional  constraints.  However,  for 
the  purposes  of  this  paper  the  original,  simpler  problem  (1)  will  suffice. 

Assume  that  the  state  equations  are  a  block  system;  two  blocks  will  suffice  for  the  points 
to  be  illustrated: 


h{x,Ui,U2) 


hl(x,Ui,U2) 

h2(x,Ui,U2) 


In  the  context  of  Multidisciplinary  Design  Optimization,  the  blocks  hi  might  describe  the 
state  equations  for  different  disciplines,  such  as  structures  and  aerodynamics,  together  with 
interdisciplinary  consistency  conditions,  and  the  aggregate  h  describes  the  coupled  multidis¬ 
ciplinary  analysis  system.  See  [5]  for  a  further  discussion  of  this  point. 

Further  assume  that  given  x  and  ui,  we  can  solve 


hi{x,ui,U2{x,ui))  =  0 


for  U2{x,ui),  and,  vice  versa,  given  x  and  U2,  we  can  solve 


h2{x,Ui{x^U2),U2)  —  0 

for  ui(x,«2)-  This  hypothesis  simply  reflects  the  independent  solubility  of  the  disciplinary 
analyses.  Note  that  this  hypothesis  means  that  the  partition  of  the  state  variables  corre¬ 
sponds  to  that  of  the  state  equations;  i.e.,  if  there  are  n,-  equations  in  /i,-,  then  there  would 
be  n-i  state  variables  in  and  the  system  solved  for  Ui  would  be  square.  We  will  also  make 
the  assumption  that  /  and  h  are  continuously  differentiable  in  their  arguments.  This  is  rea¬ 
sonable  for  many  practical  MDO  problems  that  can  be  brought  into  the  form  of  a  classical, 
smooth  mathematical  programming  problem.  On  the  other  hand,  there  remains  the  issue  of 
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design  optimization  via  the  search  for  radically  new  designs,  in  which  one  tends  to  encounter 
discontinuties  due  to  discrete  jumps  in  design  decisions. 

We  will  consider  three  possible  formulations  of  the  optimization  problem  (1).  The  first 
is  the  problem  in  its  ostensibly  unconstrained  form;  following  [5],  we  will  refer  to  this  as  the 
Multidisciplinary  Feasible  (MDF)  formulation: 

minimize  fMDF{x)  =  f{x,Ui{x),U2{x)) 

(2)  where  Ui(a;),U2(a:))  =  0 

h2{x,Ui{x),U2{x))  =  0. 

At  each  iteration,  we  feed  x  into  the  multidisciplinary  analysis  system  and  solve  for  the  state 
u.  Only  the  design  variables  x  are  treated  as  independent  variables.  This  is  an  example  of  a 
variable  reduction  approach,  in  which  we  use  the  equality  constraints  to  define  some  of  the 
variables  as  functions  of  the  others,  and  eliminate  them  as  independent  variables. 

At  the  other  extreme,  we  could  view  the  state  equations  purely  as  equality  constraints, 
yielding  the  All-at-Once  (AAO)  formulation: 

minimize  fAAo{x,ui,U2)  =  f{x,ui,U2) 

(.3)  subject  to  hi{x,ui,U2)  =  0 

h2{x,Ui,U2)  =  0. 

This  approach  has  many  different  names  in  many  different  fields.  For  instance,  it  is  known 
as  Simultaneous  Analysis  and  Design  (SAD  or  SAND)  in  structural  optimization  [11].  It 
is  with  this  view  of  the  state  equations  as  equality  constraints  that  “feasible”  appears  in 
the  name  “Multidisciplinary  Feasible” ;  at  each  iteration  we  are  feasible  with  respect  to  the 
state  constraints.  The  rationale  for  such  a  formulation  is  that  rather  than  spend  a  lot  of 
time  attaining  multidisciplinary  feasibility  far  from  an  optimal  solution,  this  constrained 
formulation  allows  us  the  freedom  to  attain  feasibility  only  at  the  same  time  as  optimality. 

Intermediate  to  these  two  formulations  is  one  that,  following  John  Dennis’s  apt  termi¬ 
nology,  will  be  called  the  In-Between  formulation.  In  this  approach  we  eliminate  one  of  the 
blocks  of  state  variables  via  the  state  equations  (say,  the  block  U2)  to  arrive  at  the  following 
problem: 

minimize  //s(a;,  ui)  =  f{x,  Ui,  U2{x,  ui)) 

(4)  subject  to  /i/B(x,ui)  =  hi{x,ui^U2{x,Ui))  =  0 

where  h2{x,ui,U2{x,ui))  =  0. 

In  the  taxonomy  presented  in  [5],  the  In-Between  formulation  corresponds  to  the  Individ¬ 
ual  Discipline  Feasible  formulation.  The  consistency  constraints  of  [5]  correspond  to  hi  in 
(4),  while  the  individually  feasible  disciplines  are  subsumed  in  ^2.  This  formulation  shares 
with  the  AAO  formulation  the  advantage  of  additional  degrees  of  freedom  afforded  by  the 
constraints,  while  having  perhaps  substantially  fewer  explicit  variables  in  the  optimization 
problem,  since  we  have  performed  a  partial  variable  reduction  by  eliminating  U2  as  an  inde¬ 
pendent  variable.  For  a  further  discussion  of  the  relative  advantages  and  disadvantages  of 
these  formulations,  see  [5]. 
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The  goal  of  this  paper  is  to  show  how  these  three  formulations,  the  associated  sensitivi¬ 
ties,  and  reduced  basis  methods  for  their  solution  are  related.  We  will  show  how  one  may,  in 
a  fairly  simple  manner,  pass  between  the  dilferent  formulations  together  with  algorithms  for 
their  solution.  It  will  turn  out  that  to  apply  a  reduced  basis  algorithm  to  the  three  formula¬ 
tions  requires,  in  principle,  only  a  change  to  one  step  of  the  optimization  algorithm.  This  is 
due  to  the  structure  of  the  problem  introduced  by  the  state  constraints.  This  structure  has 
important  consequences  for  the  design  and  implementation  of  disciplinary  analysis  codes  if 
they  are  to  be  used  in  optimization. 

The  work  in  [7]  compares  these  three  formulations  for  a  model  large-scale  parameter 
estimation  problem  and  shows  the  potential  promise  of  the  In-Between  formulation.  The 
implementation  used  in  that  work  realizes  the  idea  presented  here,  and  the  three  formulations 
were  solved  with  substantially  the  same  code,  as  will  be  discussed  in  §5. 

The  reduced  gradient  and  the  reduced  Hessian  will  be  reviewed  in  §2.  In  §.3,  we  will 
examine  the  sensitivity  calculations  associated  with  the  MDF  and  In-Between  formulations. 
There  we  will  see  how  they  can  be  obtained  from  those  of  the  AAO  formulation.  In  §4  we 
will  outline  how  a  reduced  basis  nonlinear  programming  (NLP)  algorithm  works,  and  in  §5 
show  how  one  may  pass  between  reduced  basis  algorithms  for  the  solution  of  the  different 
formulations. 


2.  The  reduced  gradient  and  Hessian 

This  section  reviews  the  notions  of  the  reduced  gradient  and  the  reduced  Hessian.  First 
recall  the  classical  implicit  function  theorem  [4] . 

Theorem  2.1  (The  Implicit  Function  Theorem).  Suppose  $  is  a  mapping  from 
an  open  subset  o/R“  x  R“  into  R“.  Suppose  {yo,Vo)  is  a  point  in  R“  x  R“  such  that 

1.  <^{yo,vo)=0; 

2.  $  is  continuously  differentiable  at  {yo,vo); 

3.  the  matrix  ^{yo,Vo)  is  invertible. 

Then  there  exists  a  neighborhood  T  ofyo  such  that  for  each  t/  €  F,  the  equation  $(y,u)  =  0 
is  soluble  for  v{y)  €  R“.  Moreover,  the  derivative  of  this  solution  v{y)  with  respect  to  y  is 
given  by 

dy  \  /  ^y' 

Formally,  the  latter  formula  is  the  result  of  implicit  differentiation  of  ^{y,v)  =  0  to 
obtain 

d^dv  _  ^ 
dy  dv  dy 

and  then  (5). 

Now,  suppose  that  v{y)  is  the  unique  solution  to  a  state  relation 

Hy,'v{y))  =  0 
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(6) 


where  $  is  twice  continuously  differentiable  and  jdv  is  invertible.  In  practice,  the  validity 
of  these  hypotheses  typically  follows  from  the  existence  and  uniqueness  theory  for  the  solution 
of  the  equation  represented  by  (6).  With  these  hypotheses,  one  may  apply  the  Implicit 
Function  Theorem  to  compute  sensitivities.  Before  doing  so,  let  us  first  define  the  matrix 


W  =  W{y,v)  = 


We  will  call  W  the  injection  operator  associated  with  $  since  it  is  a  one-to-one  mapping 
from  R“  into  x  R*^  and  is  invertible  on  its  range  (i.e.,  W  has  full  column  rank).  Its 
transpose  we  will  call  the  reduction  operator. 

Let 


and 


We  call  B  the  basic  block  since,  by  assumption,  B  =  d^{dv  is  invertible  and  hence  is  a 
basis  of  R“.  The  associated  state  variables  v  are  the  basic  variables.  The  remaining  design 
variables,  y,  are  the  non-basic  variables,  and  N  the  non-basic  block.  In  terms  of  the  basic 
and  non-basic  blocks, 


W  = 


I 

-B-^N 


Note  that  W  is  a  basis  for  the  nullspace  of  the  linearized  state  constraint  operator: 


(7) 


V$^W  =  (iV  B) 


I 

-B-'^N 


=  0. 


We  will  return  to  this  point  in  §4  and  §5. 

Finally,  given  F  :  R“  x  R“  — >  R,  define  A  by 

(8)  A  =  A(j/, «)  =  -  j  (S,,  v)V,F(y, ») 

and  the  Lagrangian  i{y,  u;  A)  by 

i{y,  V,  A)  =  F{y,  u)  +  A^$(y,  v). 

The  quantity  A  is  a  Lagrange  multiplier  estimate  arising  from  variable  reduction  [10].  For 
those  familiar  with  the  language  of  “the  adjoint  method”  in  control,  A  is  the  costate  or 
adjoint  state. 

Proposition  2.2.  Let  F{y)  =  F{y,v{y)),  where  v{y)  is  computed  via  (6),  and  F  : 

Rm  X  Rn  Rq 

(9)  =  V(y,.,)F'^{y,v{y))W{y,v{y)). 
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If  F  is  a  real-valued  function,  then 

(10)  VlF{y)  =  W^{y,  v{y))  (vfy^^^Fiy,  v{y))  +  v{y))\{y,  v{y)))  W{y,  v{y)). 

By  the  quantity  is  meant 


In  the  case  of  a  real- valued  F,  transposition  of  (9)  tells  us  that  VF  is  given  by 
(11)  ^yHv)  =  ^^(^5  ^{y))^{y,v)F{y,  u(y)). 

The  quantity  appearing  on  the  right-hand  side  of  (11)  is  known  as  the  reduced  gradient  [10], 
since  it  is  the  product  of  the  reduction  operator  W'^  with  VF.  Similarly,  the  quantity  on 
the  right-hand  side  of  (10)  is  called  the  reduced  Hessian  of  the  Lagrangian. 

Proof  Computing  the  Jacobian  of  F,  we  see  that 

dF,  ,  OF,  ,  ,,  dF.  .  ..dv.  , 

This  and  the  Implicit  Function  Theorem  then  yield 


^ivAy))  ^(yAv)) 


which  is  (9). 

Next,  the  Hessian  for  real-valued  F.  We  have 


d^\ 


-^^TvYyiyAy))^F,(yAv))j^ 


d^F  dFy  dv^ dFy  dFydv  dv^ dFy  dv  ^  <Pv 
dy^  dy  dy  dv  ^  dy  dy  dy  dv  dy  ^  dy'^ 

We  can  rewrite  this  in  terms  of  the  reduction  and  injection  matrices  as 

/  dFy  dPA  (  \ 

=  (,f)  dv  dy 

\  dy  J  ^  ^  ^  dy^ 

\  dv  dv  /  \  dy  j 


d'^F 

dx^ 
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Repeated  implicit  dilferentiation  of  $(t/,v(t/))  =  0  yields 


whence 


d^\  ^ dv^d^y  d^vdv  dv^d^ydv 
dv)  \  dy  dy  dv  dy  dy  dy  dv  dy 


Thus, 


dFfdh\  ^  fd^y  dv^d^y  d^ydv  ^dv^d^ydv 
dv  /  \  dy  dy  dv  dy  dy  dy  dv  dy 

jfd^y  dv^d^y  d^v  dv  dv^ d^v  dv\ 

\  dy  dy  dv  ^  dy  dy  dy  dv  dy  j 


Thus  we  finally  arrive  at 

^{y)  =  i^\y,v)F  +  ^ly,v)h>)  W  =  W^  (^(V,v)%>  ^(2/); 

which  is  (10).  □ 

3.  Sensitivities  for  the  three  formulations 


In  this  section  we  will  derive  some  formulae  relating  the  sensitivities  of  the  MDF,  In-Between, 
and  All-at-Once  formulations.  We  will  see  that  the  sensitivities  are  all  derived  from  those  of 
the  AAO  formulation  in  simple  and  interesting  ways. 

Proposition  2.2  says  that  the  gradient  and  Hessian  associated  with  the  MDF  formulation 
are  the  reduced  gradient  and  reduced  Hessian  of  the  Lagrangian  associated  with  the  AAO 
formulation,  as  we  will  now  discuss.  Let  Baao  and  Naao  denote  the  basic  and  non-basic 
blocks  associated  with  the  AAO  formulation. 


Baao  = 


dhi  dhi 
dui  du2 

dui  du2 


Naao  = 


let  Waao  denote  the  reduction  matrix  associated  with  the  AAO  formulation. 


Waao  = 


—Baao  Naao  )  ’ 
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let  Xaao  denote  the  Lagrange  multiplier  associated  with  the  AAO  formulation, 

XaAO  =  —Bj^Q'^ufAAOi 

and  let  Iaao  denote  the  Lagrangian  associated  with  the  AAO  formulation, 

Iaao (a:,  “w;  X)  =  fAAo{x,  u)  +  X^h(x,  u). 

Then  we  have  this  corollary  of  Proposition  2.2. 

Corollary  3.1.  Suppose  that  (x,u)  is  feasible  with  respect  to  the  multidisciplinary 
analysis  constraints:  h{x,u)  =  0.  Then  the  gradients  and  Hessians  at  {x,u)  of  the  MDF  and 
AAO  formulations  are  related  by 

(12)  ^xfMDpi^)  — 

(13)  =  WIao{^^'^)^(x,u)^aao{x,u-,Xaao{x,u)) 
and 

(14)  ^lfMDF{x)  =  W'^{x,u)  [Vf^^^'^£AAo{x,U-,  XAAo{x,u))j  WaAo{x,u). 


Proof  The  relations  (12)  and  (14)  follow  immediately  from  Proposition  2.2.  The  equa¬ 
tion  (13)  is  a  consequence  of  the  fact  that  Wjj^QS/'h  =  0.  0 

Next  we  turn  attention  to  the  sensitivities  associated  with  the  In-Between  formulation 
(4).  We  will  denote  by  S  the  Schur  complement 

^  dhi  dhi  fdh2Y'' dh2 


This  is  the  Schur  complement  of  the  basic  block  Baao  upon  elimination  of  the  variable  U2- 
One  can  easily  check  that 


dh2 


._i  dhi  ( dh2 


dh2g_idhi  / dh2 


du2  \du 


Denote  by  W2  the  reduction  matrix  associated  with  the  elimination  of  U2: 


The  next  result  concerns  the  basic  and  non-basic  blocks  for  the  In-Between  formulation. 
Equations  (15)  and  (19)  mean  that  the  basic  block  associated  with  the  In-Between  formula¬ 
tion  (4)  is  the  Schur  complement  that  results  if  U2  is  eliminated  from  the  basic  block  of  the 
All-at-Once  formulation. 
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Proposition  3.2.  The  nonbasic  block  associated  with  the  In-Between  formulation  (4) 


dhiB  dhi  dh\  / dh2\  dh2 

dx  dx  du2  \du2  /  dx 

The  basic  block  associated  with  the  In-Between  formulation  (4)  is 

dhiB  dh\  dh\  / dh2\  dh2 

dui  dui  du2  \du2)  dui 

Proof  Applying  Proposition  2.2  with  y  =  {x,  ui),v  =  U2,  and  F{x,  ui)  =  hiB{x,  ui,  U2{x,  Ui)) 
yields 

/ dhiB  dhiB\  _  ( dhi  dhi  dhi\ 

\  dx  dui  )  \  5a;  dui  du2  j  ^ 

I  h  ^  \ 

_  ^  0  In, 

y  5a;  5wi  5^2 /  f  dh2\  5/i2  / 5/^2 \  5/t2 

\  \5u2/  dx  \5u2/  5ui 

from  which  we  obtain  (18)  and  (19).  □ 

Let 


denote  the  reduction  matrix  associated  with  eliminating  ui  from  the  In-Between  formulation. 
Let 


A/b  =  — 


denote  the  Lagrange  multiplier  associated  with  the  In-Between  formulation.  In  the  In- 
Between  formulation,  we  eliminate  U2  from  the  AAO  formulation  as  an  explicit  variable; 
the  following  proposition  says  that  the  the  multiplier  Xib  can  be  obtained  from  the  AAO 
multiplier  by  dropping  the  component  corresponding  to  the  constraints  that  define  U2- 

Proposition  3.3.  At  a  point  (a;,Ui)  feasible  with  respect  to  hjB,  is.,  satisfying 
hiB{x,ui)  =  0  (and  a  fortiori,  h2{x,ui,U2{x,ui)  =  0),  we  have 

(22)  A/B(a;,Mi)  =  (/«j  0)  Xaao{x,ui,U2). 


Proof  Applying  Proposition  2.2  to  compute  V ix,ui)  fiB{x,ui)  we  have 
Xib  =  -S~'^WuJib{x,ui) 

—  S  (0  lui)^ 

—  S  (0  All )  1^2  (^i  ^15^2)' 
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Writing  this  out  leads  to 


A/s  =  —S  ^(0  lui) 


( 


\ 


0  7, 


( df^\  ^  \ 

dx  \du2 

dh2^  (dh2 


-T 


Ul 


V {x,u\,U2)f AAoiS^i  ^1)  ^2) 


dui  \du2J  / 


0  S 


-T 


du\  I  du2 ^ 


-T\ 


^  (x  ,ui  ,U2)  f A  AO  t  ^1?  ^2) 


~  {lui  ^)BaAO^q  q  j  (x,ui,u2)f AAoip^i'^U'^i) 


(7„i  0)  B/^oVJaAo{x,  Ul,  U2) 

(7„j  0)  \aao  ■ 


□ 

We  have  the  following  factorization  of  the  All-at-Once  reduction  operator  in  ternas  of  the 
In-Between  reduction  operator  and  the  reduction  operator  associated  with  the  elimination 
of  U2  as  an  explicit  variable. 

Proposition  3.4.  Suppose  that  {x,ui)  is  feasible  with  respect  to  hjB,  as  in  Proposition 
3.3.  Then 


(23) 


Waao  =  W2W1B. 


Proof.  The  proof  is  another  icky  calculation.  We  have 


Wo  = 


and 


WiB  = 


'dh2V"^ 

dx 


.du2 


-1 


\ 

dh2 

du2  ) 


so 


W2W1B  = 
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But  from  (16)  we  see  that  the  last  two  rows  of  this  product  are  none  other  than 
-Baao^aao  = 

5-1 

5^2  ^C-1 

du2 )  dux 

Th.nsWAAO  =  W2W:B.  D 

The  next  proposition  says  that  the  reduced  gradient  and  reduced  Hessian  for  the  In- 
Between  and  All-at-Once  formulations  are  the  same  at  appropriately  feasible  points. 

Proposition  3.5.  Suppose  that  {x,ui)  is  feasible  with  respect  to  hiB,  as  in  Proposition 
3.3.  Then 

(24)  Ui)  = 


^  \  /  dhi  \ 

du2  V5«2/  dx 

\du2j  '^\5u2/  dui  5u2  V5u2/  j  ^  dx  / 


IP/5  (^5  )  "^/b)  ^^AAO^ (  X,Ui,U2 

=  Hf  {y\x,u,,U2)^AAo{x,Ux,U2AAAo))  1^2, 


(27)  IP/sVj^^^)£/s(ar,ui;  A/b)IP/b  =  Wljxo^\^,ur.u2)^AAo{x,ux,U2\  \aao)Waao, 

where  A/b  =  A/B(a:,ui)  and  Xaao  =  Xaao{x,Ui,U2). 

Proof.  From  Proposition  2.2  we  have 

^{x,ui)flB  =  IPz^V  {x,Ui,U2  )fAAO', 

this  combined  with  Proposition  3.4  yields  (24).  Equation  (25)  then  follows  since  Wj^S/hiB 
0  and  Wljxo^h  =  0. 

Applying  Proposition  2.2  to  F  =  Iib  we  obtain 

„j)^/B  =  1^2^  {ylj;^ui,U2)fAAO  +  +  '^\x,uuU2)^A-^  1^2 


where  A2  is  given  by 


(0  Tuj)  (^(^x,ui,U2)fAAO  4"  ^ {x,u\,u2)hlBXlB^  • 


Using  (16)  and  (21),  we  obtain 


Ao  — 


\du2 )  ydu2  du2 

(0  /us)  {—BjXAo'^ufAAo) 

(0  Iu2 )  Xaao  • 


du\  du\  I  du2  j  du2 
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At  this  point  the  component  of  Xaao  corresponding  to  the  block  /12  reappears. 
Moreover,  (22)  tells  us  that  A/b  =  {lui  0)  Xaao-  Thus, 

=  W2  {yfx,uuU2)fAAO  +'^lx,m,U2)^^AAo)  W2, 
which  is  (26).  Moreover,  by  Proposition  3.4, 

=  ^AAO  ^  lx,m  ,U2/ AAO  WaaO  ■ 


D 


4.  Reduced  basis  algorithms 

In  this  section  we  will  discuss  a  model  Sequential  Quadratic  Programming  (SQP)  subproblem 
and  reduced  basis  methods  for  its  solution.  SQP  subproblems  similar  to  this  one  are  at  the 
heart  of  many  equality  constrained  NLP  algorithms. 

The  fundamental  SQP  subproblem  for  (3)  is 

,  .  minimize  Hs  +  g^s 

subject  to  +  h  =  0, 

where  H  is  an  approximation  to  the  Hessian  of  the  Lagrangian  and  g  is  the  gradient  of 
the  Lagrangian.  For  a  trust-region  algorithm  one  would  have,  in  addition,  some  manner  of 
trust-region  constraint,  but  for  now  this  detail  will  be  ignored.  This  subproblem  may  be 
stated  in  many  equivalent  ways;  for  a  discussion,  see  [15]. 

Now,  if  one  had  a  point  that  was  feasible  with  respect  to  the  linearized  constraints, 
and  a  basis  Z  for  the  nullspace  of  the  Jacobian  of  the  constraints,  then  one  could  express  s 
as 


s  =  -f-  Za 

and  the  SQP  subproblem  could  be  reduced  to  an  unconstrained  problem: 

(29)  minimize  Z^ HZa  -I-  (^r  -|-  Z^ cr. 

In  a  reduced  basis  approach  to  an  equality  constrained  optimization  problem  we  take 
advantage  of  a  certain  form  of  basis  for  the  nullspace  of  the  linearized  constraints  [10].  At 
each  iteration  of  the  optimization  algorithm,  we  seek  a  partition  of  the  Jacobian  of  the 
constraints  of  the  form 

V/i^  =  (A  B) 

where  B  is  invertible;  then 

2  =  (  -B-'JV 
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is  a  basis  for  the  nullspace  of 

But  we  have  already  seen  an  example  of  such  a  nullspace  basis  at  work.  In  the  case  of 
state  constraints,  we  have  an  immediate  and  fixed  choice  for  the  basis  matrix  B,  to  wit,  the 
linearized  state  operator.  This  is  the  import  of  (7).  Moreover,  the  invertibility  of  B  allows 
us  to  obtain  a  linearly  feasible  point  we  may  simply  take 

(30) 

This  corresponds  to  solving  the  linearized  state  equation,  or,  if  one  prefers,  taking  one 
unglobalized  step  of  Newton’s  method  towards  the  solution  of  the  state  equation. 

In  terms  of  W,  then,  the  model  SQP  subproblem  (28)  could  be  transformed  to  the 
unconstrained  problem 

minimize  HWa  +  {g  +  Hs^^yW^a. 

One  would  solve  this  problem  for  the  reduced  step  <t— a  step  in  the  design  variables  alone— 
and  then  take  the  step  s  =  kPa  in  all  the  degrees  of  freedom  in  the  optimization  problem, 
both  the  design  variables  and  those  state  variables  kept  explicit. 

Note  that  this  is  not  only  a  convenient  reduction,  but  in  the  context  of  MDO  this  is 
almost  surely  a  necessary  step.  The  SQP  subproblem  involves  attaining  feasibility  with  re¬ 
spect  to  the  linearized  state  constraints.  Therefore  the  SQP  subproblem  is  necessarily  more 
difficult  than  solving  the  linearized  multidisciplinary  analysis  problem,  which,  for  problems 
involving  PDE,  may  be  quite  difficult  in  itself.  In  order  to  solve  the  optimization  subprob¬ 
lem  we  must  exploit  the  structure  of  the  problem — in  this  case,  its  relation  to  the  state 
constraints — and  avail  ourselves  of  the  pieces  available  from  the  solution  of  the  analysis 
problem. 


5.  A  simple  way  of  implementing  reduced  basis  algo¬ 
rithms  for  the  different  formulations 


In  this  section  we  will  see  how  the  relations  between  the  different  formulations  in  §3  mean 
that,  in  principle,  it  is  possible  to  implement  an  optimization  algorithm  for  the  A  AO  formula¬ 
tion  that  with  a  single  modification  becomes  an  algorithm  for  either  the  MDF  or  In-Between 
formulations.  Such  an  implementation  has  been  done  for  the  work  described  in  [7]. 

The  key  to  such  an  implementation  lies  in  the  relations  that  were  presented  in  §3.  To 
make  matters  concrete,  consider  the  following  trust-region  algorithm  for  large-scale  equality 
constrained  optimization  problems.  This  algorithm  was  used  for  the  work  in  [7]  and  is  a 
relative  of  the  algorithms  described  in  [3]  and  [6].  The  subscript  and  superscript  “c”  denote 
quantities  associated  with  the  current  iterate,  and  the  subscript  and  superscript  “-1-”  denote 
quantities  associated  with  the  putative  next  iterate,  following  [8]. 

At  each  iteration,  we  approximately  solve  the  subproblem 


(31) 


minimize 
subject  to 


\s^Hs  -I-  g'^s 
'^h^s  +  hc\\<e 

Ih  II  <  ^ 
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where  H  is  an  approximation  to  the  Hessian  of  the  Lagrangian,  g  is  the  gradient  of  the 
Lagrangian,  S  is  the  current  trust  radius,  and  6  is  chosen  to  enforce  Fraction  of  Cauchy 


Decrease  on 


VFs  +  he 


over  a  trust-region  of  radius  r6,  where  0  <  r  <  1  is  fixed. 

The  approximate  solution  of  the  subproblem  is  eflFected  in  two  stages.  The  first  is  to 
take  a  step  in  the  basic  variables  to  improve  linear  feasibility,  rather  as  in  (30): 


(32) 

where 


„LF 


=  ad, 


0 

\ 

fdh\ 

-1 

L 

IXq 

/ 

The  quantity  a  is  chosen  to  solve 


minimize  | 
subject  to 


a'^ h'^ d  -f"  he 
II  ad  II  <  rS. 


The  second  stage  is  then  to  improve  optimality  subject  to  the  constraint  of  not  degrading 
the  improved  linear  feasibility  achieved  by  Let 

0  =  Vh^s^^  +  he- 

The  subproblem  we  solve  to  improve  optimality  subject  to  improved  linear  feasibility  is 


minimize 


\s^ Hs  -f  g^s 
subject  to  VhFs  he  =  Q 

II  s  II  <  S. 


(33) 


This  we  do  by  using  the  reduced  basis  and  writing  the  desired  step  as 

s  =  s^^  +  Wa 


and  solving  the  trust-region  problem 


(34) 


minimize 
subject  to 


\a^W^HWa  +  {g  + 


Wa 


<6. 


We  then  test  the  aggregate  step  s  using  the  augmented  Lagrangian 

HrA,p)^f{y)  +  ^^h{y)  +  ^\\  %)  f, 

as  the  merit  function  with  the  penalty  weight  p  updated  as  described  in  [9].  Figure  1  contains 
a  rough  outline  of  the  algorithm  for  the  A  AO  formulation. 

Now  suppose  that  one  modifies  this  algorithm  by  adding  an  analysis  step  in  which  given 
design  variables  Xe,  one  solves  the  multidisciplinary  analysis  problem  h{xe,Uc)  =  0.  This 
single  modification,  illustrated  in  Figure  2,  yields  a  trust  region  algorithm  for  the  MDF 
formulation. 
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Initialization:  Choose  an  initial  (xc,Uc). 
Until  convergence,  do  { 

1.  Compute  the  multiplier 


2.  Test  for  convergence. 

.3.  Construct  a  local  model  of  £aao  about  (xc,Uc). 

4.  Take  a  step  to  improve  linear  feasibility: 

f  ^ 

=  a 

5.  Subject  to  the  improved  linear  feasibility,  take  a  step  to  improve 
optimality: 

minimize  ^Cr^WjAO^AAoWAAOf^  +  {dAAO  +  HaAOS^^)^WaaO^ 
subject  to  +  WaaoC’’  <  S. 

6.  Set  s  =  {sx'iSu)  =  +  WAAOf^- 

7.  Evaluate  (a:+,u+)  =  (xc,Uc)  +  (-Sar, -Su)- 

8.  Update  (xc,  Uc), 


Fig.  1:  Reduced  basis  algorithm  for  the  A  AO  formulation. 


Theorem  5.1.  The  modified  algorithm  in  Figure  2  is  equivalent  to  a  trust  region 
algorithm  for  the  MDF  formulation. 

Proof  We  will  check  the  following: 

1.  At  every  step,  we  are  feasible  with  respect  to  the  constraint  h{xc,Uc)  =  0.  This  is, 
of  course,  true  by  design. 

2.  The  subproblems  we  solve  in  the  context  of  the  AAO  formulation  correspond  to  a 
trust  region  subproblem  for  the  MDF  formulation. 

3.  The  merit  function  we  use  to  evaluate  the  step  taken  for  the  AAO  formulation 
corresponds  to  the  correct  merit  function  for  the  MDF  formulation. 

Because  we  solve  the  analysis  problem  h{xc,Uc)  =  0  at  each  iteration,  he  =  0  and  so 

=  0.  The  subproblem  we  solve  to  improve  optimality  then  becomes 

minimize  \  ^aao  Haao  Waao  ^  +  9aao  ^aao*^ 
subject  to  II  Waao(^  II  ^ 
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Initialization:  Choose  an  initial  Xc- 
Until  convergence,  do  { 

0.  Analysis:  Solve  h[x^u{xy)  =  0  for  the  state  u(x). 
1.  Test  for  convergence. 


7.  Analysis:  Solve  k(x^,u+)  =  0  for  w+(a:+)  and  evaluate  (x+,u+). 


} 


Fig.  2:  Reduced  basis  algorithm  for  the  AAO  formulation  with  a  multidisci¬ 
plinary  analysis  step. 


However,  in  light  of  Corollary  3.1,  this  is  none  other  than 

minimize  Icr'^VlfMDFcr  +  ^ 
subject  to  II  ||  < 

This  we  recognize  as  a  trust-region  problem  for  the  MDF  formulation,  which  establishes  Point 
2.  Note,  however,  the  non-standard  scaling  of  the  trust  region  constraint.  As  discussed  in 
[14],  this  scaling  is  quite  natural  since  it  measures  the  effect  of  changes  in  the  design  variables 
on  the  state  variables. 

Moreover,  Point  3  holds.  Because  h{x,  u)  =  0  at  each  step,  the  augmented  Lagrangian 
merit  function  reduces  to 

fAAo{x,u{x))  +  X'^j^ji^Qh{x,u{x))  ^  II  h{x,u{x))  11^  =  fAAo{x,u{x))  =  fMDF{x), 

which  is  the  merit  function  for  the  unconstrained  MDF  formulation.  Thus  we  have  a  trust 
region  algorithm  for  the  MDF  formulation.  □ 

Thus,  simply  by  the  addition  of  an  analysis  step  the  reduced  basis  algorithm  for  the 
AAO  formulation  becomes  a  reduced  basis  algorithm  for  the  MDF  formulation,  without  any 
re-implementation  of  sensitivities  or  other  procedures.  In  fact,  this  reduced  basis  algorithm 
is  an  instance  of  a  class  of  algorithms  known  as  Generalized  Reduced  Gradient  methods, 
which  date  back  to  Abadie  and  Carpentier  in  the  early  1960s.  For  further  discussion  and 
illustration  of  these  methods  applied  to  problems  with  state  constraints,  see  [1,  2, 12, 13, 14]. 

Now,  what  happens  if  one  adds  a  partial  analysis  step  to  the  algorithm  for  the  AAO 
formulation,  an  analysis  that  enforces  feasibility  with  respect  to  the  block  of  constraints 
h2,  as  in  Figure  3?  As  it  happens — as  it  was  meant  to  happen — we  obtain  a  reduced  basis 
algorithm  for  the  In-Between  formulation. 


15 


Initialization:  Choose  an  initial  uj). 

Until  convergence,  do  { 

0.  Analysis:  Solve  h2ix,ui,U2{x,ui))  =  0  for  U2{x,ui). 

1.  Test  for  convergence. 


7.  Analysis:  Solve  hi{x'^ ,uf,U2)  =  0  for  and  evaluate 

(a:+,u+). 


} 

Fig.  3:  Reduced  basis  algorithm  for  the  A  AO  formulation  with  an  analysis 
step  that  eliminates  U2- 


Theorem  5.2.  The  modified  algorithm  in  Figure  3  is  equivalent  to  a  reduced  basis  trust 
region  algorithm  for  the  In-Between  formulation. 

Proof  We  will  check  the  following: 

1.  At  every  step,  we  are  feasible  with  respect  to  the  In-Between  state  constraint 
h2{x,ui,U2{x,ui))  =  0.  This  is  true  by  design. 

2.  The  subproblem  we  solve  in  the  context  of  the  AAO  formulation  to  improve  model 
feasibility  yields  a  similar  step  for  the  In-Between  formulation. 

3.  The  subproblem  we  solve  in  the  context  of  the  AAO  formulation  to  improve  op¬ 
timality  subject  to  improved  model  feasibility  does  the  same  for  the  In-Between 
formulation. 

4.  The  merit  function  we  use  to  evaluate  the  AAO  formulation  corresponds  to  the 
correct  merit  function  for  the  In-Between  formulation. 

We  have 


5-^ 

du2 )  dui 


du2  \du2) 

\du2j  \5M2y  dui  du2 
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lui 

\ 

(dh2\-^ 

dh2 

Bjb^ib 

\du2) 

du\ 

) 

and  hence 


.LF  _ 
^AAO  — 


-aBA\oh 


'dho 


-I 


*-Ul 


duo 


dh2 

dx 


.duo 


-1 


dx  j 


^LF 


where  a  is  chosen  so  that 


.LF 

^IB 


W2 


h"24I 


=  rS. 


This  settles  Point  2.  Again,  note  the  non-standard  scaling  of  the  trust-region;  this  tracks 
the  effect  of  the  step  on  the  implicit  variable  U2. 

Meanwhile,  by  Proposition  3.5, 


Waao  = 

^AAO  HaAO  ^AAO  = 
"^aaoSib  = 


IV2WIB, 

WJsHibWib, 

W'IbQib, 


and 


Wi^oHAAO^AO  =  WisW‘ HAAoW,si^ 


LF 


=  WfsHiBS^^. 

Thus  we  have  the  equivalence  between  the  subproblem 

minimize  \cr'^W^AQ  Haao  Waao  <x  +  Haao  sTao  +  Q^ao  ^ 


^A^O  "b  WaA0<^ 


<  S 


subject  to 

for  the  AAO  formulation  and  the  subproblem 

minimize  Ict'^WIbHibWibo-  -|-  Hibs^ao  +  9Jb<^ 
"  +  WiB<r 


subject  to 


W2 


<6 


for  the  In-Between  formulation,  which  establishes  Point  3. 

Moreover,  because  h2{x,Ui,U2{x,ui))  =  0  at  each  step,  Proposition  3.3  tells  us  that  the 
augmented  Lagrangian  merit  function  reduces  to 

fAAo{x,  u{x))  +  XaaoH^^  S  II  h{x,  u{x))  f  = 


fiB{x,  ui)  -f  X^BhiBix,  ui)  -f  J  II  hiB{x,  Ui) 
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which  is  the  augmented  Lagrangian  for  the  In-Between  formulation,  confirming  Point  4. 
Thus  we  have  a  reduced  basis  trust  region  algorithm  for  the  In-Between  formulation.  □ 
This  shows  that  the  addition  of  an  appropriate  analysis  step  to  the  reduced  basis  algo¬ 
rithm  in  Figure  1  for  the  AAO  formulation  yields  reduced  basis  algorithms  for  the  MDF  and 
In-Between  formulations.  Similar  results  hold  for  other  algorithms  one  might  consider,  such 
as  an  SQP  approach  using  a  line-search. 


6.  Conclusion 

The  results  of  §3  and  §5  follow  from  structural  features  of  optimization  problems  governed  by 
state  constraints.  A  number  of  other  practical  consequences  of  these  structural  features  are 
discussed  in  [14].  The  results  presented  here  illustrate  some  of  the  rich  and  useful  structure 
of  optimization  problems  with  state  constraints. 

This  paper  has  demonstrated  that  a  properly  structured  and  implemented  reduced  basis 
algorithm  can  be  modified  in  a  conceptually  and  practically  simple  manner  to  produce  al¬ 
gorithms  for  alternative  problem  formulations,  demonstrating  a  truly  gratifying  connection 
between  the  formulation  of  a  problem  and  algorithms  for  its  solution.  One  can  easily  build 
a  telescoping  family  of  algorithms  that  correspond  to  different  formulations  of  the  problem, 
and  have  different  computational  behavior. 

The  key  to  this  approach  is  the  availability  of  certain  components  inside  the  state  con¬ 
straints;  in  particular,  at  the  very  least  the  actions  on  vectors  of  the  operators 

dx  ^  du\du  j  \du  j 

If  simulation  codes  were  written  to  make  these  components  accessible  to  those  of  us  doing 
optimization,  then  our  task  would  be  greatly  facilitated,  and  alternative  formulations  and 
algorithms  could  be  easily  explored. 
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